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This paper is directed at those persons -interested in tho manner in 
which a program may be developed'using a field t£enLer(Bd, competency based/ 
education model or ^ho might wish to consider adaptJLng some of the' essential 
features of the niodel to ^ther instructional programs. The model was developed 
originally for the preparation of administrators under funding from the* 
United States Office, of Education and the Bureau of Educatioji for the Handf-*. 
capped . - • ^ * 

^ A systems approach will be used to clatify the basic phases or components 
involved in program "developmenC which will be discussed along with example^ 
of specific, program procedure's and instrumentation. The modeT.lng language 
us^d to ill^strat-e the model was developed by Silvern* (1972) and shows the \ 
flow of informatioI^\; thi^ough the System. Systems models of this type 'are 

- . ■ . ■' . • 

intentionally de-^feloped at a general lev^l and never change, Cheir major 

^ » • » *' * 

ejLemen£s and relationships during imgJ,ementatlon. Focus on inpu.tsV 'activist i^s, 

' ' \ . ' ■■ . ' 

and outputs has the advantage of being relatively independent, of content; and 

a program stated in systems terms can be more readily adapted ta any field 

* - / 

in whic^,. similar conditions obtain 'initially (i.e. where 'performancjS can be 
observed) . " ' . o - . 

Overview " • ' ' \ ' ' ^ 

f' • * " 

In its, most general form a field cfentered^ competency based, education 

model' is relatively straightforward and has many features in common with 

' " ^ ' ' " ' . 

other competency based ^education models. ItJs characteristics, corollaries 

. ■ • {• \ 

and '.assumptions are presented in outline form in Table 1. 

■ ■• ^ \ . . ^ ^ 

The basis for evaluation of a* program developed using the model is each 

i * 

individual's learning and performance. A primary premi<se of this model jLs 
that there is a direct relationship between training .pffered and inethods 
of evaluation. However, no, attempt is made to show the effects of such a 



•preparation program on student (child) ox client performance. One reason 4* 
is"T:hat the effect^ of staff 4evelopment on children progress still is not - 
clear and is a topic thad.^enerateS Considerable controversy withirv ^competency ^ 



based, .teacher educ'atfbn. • Fhrthisr; 4th^ere ^Is little reason to b,elieye that a 

^ ^i^^ct result o^ administrators preparatidn'will be seen through impro/ein^nt 

in child le,aming'.even,^l^ugh pupil, giowth is't^ie purpose of all school-. 
» - \ . . - ■ ; 

^' related activities. , • c 
. V . ^ - ' Insert Ta^lfe 1 here . . f . 



Figure 1 'indicaites the sequence'of dWelopWental acti^^Ttties. First, » 

' ' : ' • ' ' . ^ ^ • '^'A ' : 

the position or group of persons for whom an educational program is to be 
• developed 4s specif i^c and its characteristics and tra^.ning nee^s are described, 
/ Second, the iiompetenaies which 'persons in tHat position shou;Ld obtain are^. 
/• identified, .based- on characteristics of the position itself. Preparation of 
ipstruction and develDpm mt of an assessment system proceed concurrently 

o .' ' ■• ■' 

since these 'two activities are interdependent," However, as the feedback 
'arrowfe indicate, assessment affects the instruction component of the model. 
]^±th the possible exception of initial designatio^' of the population to be 
trained, development and modification of thfe training program are based^ upon 
objective data to a larger degree than is usual (Heath and Nielspu^ 1974); 
aftd most data management is computer J)^ed. < . , - 



Insert Figure 1 h6re 



,Ta6le 2 provides a further overview of the model by listing major program 

it - ' ' • 

development q^uestions to be answered for each component of the model, Satis- 



factor} decisions in response to^eacfj^c 



uestion n]ay be considered|ii:o be the 



♦ • 
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• , Table 1 
Field Centered; Competency" Ba^ed Model; 
fpr 'Program Development tlharacteristitts 



• « 



Chasacteristics 

• ft 

^ 



Corollaries ^ 



Assumptions 



1. Goa]^s df tlie training * 
program are stated \i' 
as competencies bv 
performances ^ 

2/ Performances are 
^derived empirically 
from job " * , 



3. Cpre^of tWlnimina 

essential competencies 
will he taughtis^ 



4. Instruction 'is ba^ed 
on individual needs 



'5. Instruction is 
field-^entereci 



S 



6. Kinds of instruction 
taught — facts , concepts 
and* skills 



7. • Evaluate training 
^jirogram by student 
learning and perfor- 
mance , 



Traljfiing cojitenjtrxmd* 
*p^erf ormance crit?eri^ 
will change over .time 
as. does job » 

Other performances may 
be desired for specific 
positions 



Amount and conten't^will 
vary. Ra4?e cf progress 
will vary' i * * 



E^faluat(e by demonstrated 
retention of ±nforii\ation 
and perf(5rman^'- (Actual 
"of simulated,) of Skills 



t5 



R<etlevant goals can , 
be identiified &nd so \ 
stated ' \ ' 



,This is a teasonable 
preparation base 



Those skills can-^be 
-'agreed upon. Persona 
with these tore skills, 
can function in entry 
leVel positions 

Varying levels of 
prior training, 
experience, and ability 

Continuing education ^ 
for employe's 'persons. 
Location is appropriate 
to the training to be 
, offered 

Person Ccin be success- 
ful^in job ir he/-fihe 
has those' skills and 
^ that knowledge 

There /is that direct' 
rqlat lonky ip . Can ' t 
show effects on child. • 
student/or^ client i 
perf oriftanca • ' , 
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'g<^ls.,pf that component. Table 2 also listis data-spuxces^bhat can provide 
b^ses upon which pro&ram^managers ma^ make rgg:lonal dec*is|.pns. * * 

^ Insert^Table 2<^H^rc \ / % " \: 



Nae4s Assessiaenb j/^,* -.^-^ 7 z f \ K\ ' * 

The first program development* ^asK yepTiJ^spnted. in Figure 1 component 

1.0 to identify the ,target position^, to eS4;^^mate" tha extent pf need for 

training withjn this target |)of)ulation and to des<iribe" the 4)opuXation. . , 

For some 6ducadion programs, "Purveys of neecls/for preparation programs 

c . « 

m^y tend to be bypassed UUe' to legislajiive mandate or other external directives. 

For other^ pro^ij'ams thei;e,46uSt jeither be demonstration of\tFie. training to 

, -'^ >a ' ' ^ ^ 

^ecure fuhding or incorporation of the program as an ^ongoing function of 
tUniv.ersity sel^f-examinationjand renewal processes* * ^ ^ 

Adoption of a competent)' based ^appiroac.h implies Chat definitions of. need 
for preparation programs are derived from a description of the population to 

^' . ' ■ 

be t-rained. Although ''ofej^^.dusly desirable, internal cons<:n6us am^ng faculty ^ 

is pot regarded/ in the model, as sufficient to establish needs withou^ supportihg 

- ^' * * . 

documentation obtained} froni.the field. Information 'gained from the initial 

planning phase is useful in delimiting the content and determining organization 



of instruction. 

The model under consideration in this paper wab developed and implemented 
ov^r a period of 18 months as a component of a project 'funded by the United 
States Bureau uf Education for the Handicapped to train administrators for, 

\ • ■ \ 

Sr^ecial education programs using a field centered approach. Using that pai-tic- 
ular application of the model as an example, the .following steps were taken. 

A numbei/ of** previously available sources of^ information were' used to. 
delineate the pcj^ulation .to be educated. A review of the literature yielded 



f 't 



Field 



Table 2.. . 

Centere<f, Competency^ Based Education Model 

■ c ' ■ • • : • 



Cdfiponent. 



. Major Questions i Goals ) 



SoLir,cfS";of Information 



1.0 Identify target " 
. gx)(yufat.ion/posilitfn. 



• » 



, ■ 2.0 Identify 

competencies 



3.0 Pr?pare instruction 
( materials, . instructors, 
logistics, etc.- ) " 



Is tlierfi a need for training ? 
Wlio sho.ulo 1)0 trained ?' 

, How can !hese persons . 
. . be described •? 



What do position incumbents 

have to do ? 
Wtiat do they need to know in 

order to perform adc-quatoly? 

. hft)w should content be Limited ? 

Hov/ should instruction J)e 
organized ( service delivery ■) ? 

Wh at jiia t er iai s /s t rat egi es ' f.o r 
learning'ar'e available or- 
need to be developed ? 

Who should provide .instruclia^ ? 

What instructional comppnerits 
or other factors influence 
probability of reaching 
training objectives ? 



Legislative (or other) mandate.^ 
or preferences 
.. Literature in tlie field . 
• Peqirt'Sjs. made to training 
institution 
Needs assess'rijnt (e.g. , demand 
■ personnel, j5r«ont trainiiuj • 

■levels of possiblf-_targ(.t 
- populations) - 
Data from prior training 

Goal analysis- 
Job analysis ; 
Anthropologjidil field study 
Data from^Mior trainin.g 

Judgements 'Of professionals 
in (he field, instructors, 
and participants 
Literature in the field 
' Data from prior training 



4.0 Develop assessment 
system 



What courses should 

participants takef? 
What changes in co*i?)etencies 

occur during and after 

instruction ? 



Pre and post domain-referenced 

testing 
Pre and post -performance 

assessment 

Performance 6n course 

malf rials exercises 
Data from pi/ior training 
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s^ummarles of the typical pfeparation and experience backgrouhd of Mintjesota 

special education directors .(e.^g, Spriggs, 1972.; Bilyeu, 1973;, and Wedl, 1973) 

which, along with a review of^ presently available educational opportunities, 

Suggested that priority ,be given to expanded and improved preparation in 

administrative, gkills for p.resent incumbents o^ these, positions. As the 

model is implemented, "^ata from prior, training efforts provide additional 

* * ' •» ^ 

soui^ies' of information' regarding the target population to be trained. 

Other prdgrams may wish tb, use similar .means or may rely on such* strate- 

gies a^*^emo graphic studies, internal and field surveys* or Delphi probes. 

^• / • V ■->'. 

Competency Identification 

• A second component of the' prbgram model is the process by which compe-*' 
tencies or desired performance^ are identified. A multidiscipli-nary approach „ 
is used employing three strategies (goal analysis, job analysis, and an 
""anthropological field study) which are used to cross-validate each other. 
Each of these strategies has previously beeh used alone^as the basis for 



performance specifications. Together such strategies present a reliable and 
more valid description of the minimum essential performances for a particular 
position, especially when viewed from the extent to which these s^trategies 
duplicate each other. 



4 



The specification of competencies or performance criteria is illustrated 
in Figure 2 and each of t;he technicvies employed is described below. 



Insert Figure 2 here 



Goal a nalysis . Goal analysis is Mager's (1972) procedure .for\obtaining 
consensus among a group of people and includes the following steps: First, 
a panel is '^elected, descriptive words and phrases are elicited from each 
panel member, and all responses are recorded, the 'panel then meets to edit 

^ ■ 10' . • - ■ ' 
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the list. Members ellmlriate duplication and non-essential items, fill in 

deficient areas, and rex^rite the list in performance term's. The group then 
» » * 

rates each item for desired level of performance, specifies the importance 

^ . • * 1 

or centrality of achievement at the task^ ancL agrees to the accuracy of the 

resulting material after it ha^ been edited into correct statements of 

behavioral objedtives. * . " ' 

r m 

Goal analysis prpvides the general statement of performaitce which, when 

combined -with the specific skills, tasks and knowledge* from the position 

i . . X * ; ' 

(job) J, analysis, allows, relevant, behaviorally stated objectives to be developed 

* * * ^ . * .V 

for the posltio.T being studied. Crucial ta effective goal analysis is the 

composition of the panel. In th^. case of the first application of the model, 

the special education administrator's job^ functions (as determined by literature 

review) uere divided into three parts-rfiscal management, personnel manapement., 

/ 

and special education program development— :and separ2* : panels were convened 

«/ . * ' 

for each function. This attion permitted selection of specialists in specific 
areas to participate as panel members without making each group unduly large* 

4 ' , 

Each panel included representatives f rom. lo.cal school districts, regional 

consultants. State Department of Education staff, and professors of educatior^al . 

\ 

^administration and* special education. 

Job analysis . Job analysis, the, second strategy, uses a setf of pyocedurW 

derived from industrial psychology for careful study of a :|*ob within an 

organization. It has been defined by tTie United, States Bureau of Bnployment 

Security (1965) : • * - . ' ' ' . 

The process of identifying, by observation. Interview and 
study, and of reporting the significant worker, a'ctivl ties and 
requirements and the technical *and environmental facts of a 
specific job.- It is .the identification of the tasks which ^comprise • 
. ' the job and of the skills, knowledges, abilities, and responsibilities 
that are requi;:ed of the workfer for successful performance and 
that differentiate the job from all ot>>ers (p. 5>. ' " - . 



10 

p 

a, 

^ 3"/ A number of different methods; may be employed for conductipg a job 
* analysis .^<CHese Include questionnaires and checklists, observation, Indiv- 
idual or "group Interviews, logbooks, or judgments about good and poor job 
performance, 

. \ Continuing to use th§ example, of the Special Education Administration 

* \ y ' > * , 

Training Program^^^sevi^rai studies of special education administrators used 
analysis of existing job descriptions and self-reporting by* questionnaires 
y sent to directors. : These job analysis procedures were supplemented by 

direct-pbserVi^tion and structured interviews with a small stratified sample 

' • w 

of the P0P^*ls^i22^«^^^^^'^^.^?' skills, ^and knowledge reported hy any of these 

means ^ejbe summarized, distribjited to all directors for^comraents, and modified ' 

as neededi. . The resulting position description is contained in the final 

report of thafc^job analysis (Harpiz, , 1974) . * 
« • t \ 

Specificity and In^usiveness characterized the ,dif f erences between 
rfesbifs of goal analysl^^and those of job analysis. Results of the job 
analysis included a lengthy enumeration of all the specific tasks which, 
every itipxx^sota director performed. On the other^ ^and, goal analysis included 
*j judgments' of centrality, or importance of more "global" performances and may 

have omitted some tasks entirely. The two procedures wer^ used to check 

' / • ^ »^ ' 

each other and to produce a moire accurate description. 

Anthropological field study . Data from anthropological study 'are 
^ interided to Im'prove further the validity of performance specification by^ 
\ identifying* minimum essential performances which substantiate thise. previously 
/ identified or which may have been earlier bverlqcked in tHe .goal analysis 
and job analysis. This Approach tends to produce^data not readily available' 
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from other sources, sifcH a§ the annual cycle of activities of a position 
liolder, xnf9rmation sources, ,and decision making ptocesses. Such an activity 

. ' 13 



/ 

documents the vast numbers and types of community ^and state agencies and 
admlnlstratprs and staff with whom the director .communicates both routinely 
and occasionally. It also assesses the injfluences of different organiza- 
tional structures upon the role of the position holder. 

An •anthropological study utilized ethnographic techniques and systems 
analysis. Participant observation in the form of part^^ipant-as-observer 
(as used by Harty Wolt:ott in his 1973 study of an elemeittary principal) 
provides the methodological base, ^pplemented and cross checked by several^ 

ojrhef methodologies^* present and past logs kept by position holder, time ^ 

. V ^ , 

study, intery,±ews, information orr the position holder^s calendar, and * 

drawing of decision-trees. For the first application of the model, each of 

/ 

three, position holders in three representative communities (urban, rural and 

/ 

Tniral-urh^n),^ representing three different types of administrative units 
(sipgle. school district, "^operative in ai^ Educational Service Area and 
cooperative not in an Educational Service Area) were^ studied one Y^ek ^ach 
month for one year. ^^^\^J^ 

Data from such anthropological study have multiple uses.. The study 
Should be begun .well in £idvance of program implementation to be used in 
conjunction with the' goal analysis and job analysis for initial competency 
specification. If carrieji.' out concurrently with instructional program 
activities, such study serves to rejfine or modify initial performance state- 
jnents. In either/case anthropological data are useful in setting up a 
framework within which simulations, course e;;ercises, or test questions can 

be devised. - / 

"/ 

Revalidation of competencies . It is recognized that position. require- 

/ . ' 

ments have a tendency to change over time. In most posltions^ob requirement 



and competencies will not show substantial differences over time»spans of 
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less than- three years. Consequently, the model repeats at. three year. 
Intervals those goal analysis and job analysis procedures to revalidate 
performance specification as duties and competencies change. The anthro- 
logical study will also, be repeated. 

. Preparation 'of Instruction 

. Chronologically, components 3.0 (Prepare instruction) and 4.0 (Develop 
assessment pi:pcedure) are concurrent procedures, and both should begin as 

«r * 

soon as initial competencies have been identified teqtatively. Indeed, the 
nature of the interrelationships between these two tasks suggests that 
development of performance measures might precede curriculum and other 
' instructional preparation* 

As may be seen in Figure 3, preparation of instruction, begins by 
delimiting the curriculum in view of priorities established in the goal 
analysis, job analysis and anthropological study, available information on 
present competency levels of the target population (such as preliminary 
assessment results), and pragmatic considerations such as* time, extent of 
funding, and other resources. Once the scope and sequence have been dete*:- 
mined, course preparation begins by selection of course authors who are 
specialists in specific content areas. These persons are provided with 
course objectives (the outcome of the competency identification process) 
and with questions from the domain referenced test which pertain to those 
objectives when available. 



Insert Figure 3 here 



As may be noted from Figure 4, during the developmental phases course 
authors are responsible for selecting and/or writing appropriate reading 
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ma'terlals and-fop: preparing e:^ercises on e^cl/ phase of the content area to 
give participants an opportunity to practic^ thB skills being taught and to - . ' 
apply concepts which have been presented. Course authors, also have"' a con- 



tinuing function. During the operation of training, they evaluate pexformance 
on the course exercises and thus provide participants with feedback on the 
extent to which concepts and skills have .been mastered. 

' 't 

^ — > . 

Insert Figure -4 here 

0 

Parallel to the development of curriculum ahd materials is development « 
of the service delivery system. Needs 'assessment data on the pqpulation to 
be trained and known parameters of the content of training provide some., 
clues to delivery requirements which must be met.. and needed resources. 

Assessment* . - * . 

The fourth component of the development model is one of its most important 

features. As a competency or performance based program, the field centered 

competency based education model is by definition a data-based system: 

Assessment lies at the heart of PBTE. Goals of instruction 
. must.be stated in assessable terms; learner performance, must be 
assessed and reassessed throughout the instructioflal process; 
evidence so obtained must be used to evaluate the accomplishments 
of the learner and the efficacy of the system. Remove assessment 
from PBTE and -all that is left is an enumeration of goals and 
provision of instruction which hopefully will lead tt their attain- 
ment — not much on which to pin one's hopes for significant improve- 
ment in an educational program (AACTE, 1974, p. 18). 

The model's emphasis on Assessment serves two major purposes. First, it 
enables program managers to determine on an ongoing basis the extent to which 
participants achieve the program objectives at criterion levels.. Second, it 
permits objective determination of the appropriateness of instructional 
methods, content of instruction, and established criterion levels for achieve- 
ment ^ , ^ , • , ' ' 

17 



The model focuses on tiiose competencies necessary for performance on i 
the job and, thus, employs two basic strategies to determine the extent to 
wh;Lcb these competencies are attained, first, performance assessment uses 
, simulations of actual tasks which all position holders y?iust /be? able to 

* * • ■■ / 

perform. Second, cognitive assessment measures of the kno\>fl.edgo which each 
participant must have in order to perform cssentia.l job task^. Both measures 
are obtained on a pre arid post test basis. * . v 

Other data are less formal and are collected.Jit Ararious points^ptior to, 
during, and following instruction. Such data include information rega/ding 
participants' perceptions of their competencies and the training they are j 
rex:eiving, results of course exercises completed ia the field, arjd reactions 
of field consultants in a positibn to observe participants' work. \ 

. • \ 

Complete descriptions of all instruments, subjective .and objective data 
collection procedures, and methods of analysis for the first ^application of 
the 'model may "be found in the project's evaluation design (Deno, 1974). 

The following discussion focuses on those procedures and instrume^xts 
which provide objective evidence of competency levels. Figure 5 indicates, 
the steps for developing and modifying procedurjes for cognitive and performance 
assessment of competencies. A more detailed explanation and flow chart for- 
.the assessment, system' as Implemented may be; found in llendrix (-1974). 



Insert Figure 5 here 



Performance assessment . Performance assessment consists primarily of a 
series of job tasks, derived from program', objectives, performed in simulated 
settings which approximate field conditions a,nd rated by experts for adequacy.^ 
In addition, participants' self-ratings of perceived proficiency levels are 
compared with observed levels obtained from the simulated tasks. The parameters 
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of performance , assessment are outlined briefly in Figure 6. Performance 
assessment by means of structured observations of participants' actual 



performance on the job was investigated* in the first applicrvtion of the. 
model but was discarded as not feasible due to h-igk cosrts. 



Insert, Figure *6 here 



Simulations developed for* use ^ri that applica^tiofi af the model were 

tailored to 'the specific situations whic^i an administrator would encounter. 

a > , ^ > V 

For example, the SEASIM (Special Education Administration iu Monroe City) 
/ ^ »• ^ * • A * 

simulation materials (UCEA, 1973) which ari related to'* program objectives 

were rewritten to apply to rural and multi-district progr^m^^C^ llowever, in 

many cases no materials were available; and the simulations had to be 

developed by project consultants and staff. ^ 

'\ , 

/ 

Initial performance assessment (simulations and self-ratings) usually 
t9ok place ^t a pre-instruction workshop. Participants were proVided. with* 
all necessary material^ and could complete tl.e. assessmenty.n approximately ^ 

one and one-half days. 

' * » * 

Each simulatioa for the model's first application was rated by five 

#. • 

raters: two representatives' of the existing special education administration 
^ training programs (degree programs) in the state, an experienced local 
special educatiori director, a regional consultant, and a State Department 
of Education special education representative. The current presxient of the 
state special education administrator's association was always inGJLuded as 
one of the practitioners. Raters woriked independently of each other, and 
the identities of the other participants were i[*ot' disclosed to them. Each 
simulated task was rated "pass" or "not pa^ss^, according to each rater's 
judgment. The majority opinion (three out of five) determined a participant' 
score. • 
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* • * • 

This- use of ainulatlon?; as an assessment tool departs from standard 
proci^dures sinc^ r:jost training programs follow simulations with immediate 
instruction to injprovo pe.rf ornance,. Instead,, the model uses performance ^ 
asosssment tc sf^lect areas in which instruction is to be provided* During 
* the field e:<perJcn-<^R of participants feedback on simulated performance 
• and further practice, on those tasks through course exercises assist in 
improving performance in deficient "areas. 

Following instr<Jction, participants are again assessed in those /areas 
in which they were prev;jLousiy deficient" to determine the d'egree of improve- 
ment# , " 

Cognitive assessment . 'Many objectives specify tasks which participants ^ 
P will become able to perform*,^ and demonstration of competency is complete and 
direct (performance of the 'task in a setting which simulated actual working 
conditions). ^Measurement: of these tasks may be considered |o be criterion 
referenced. However, the knowledge (information, grasp of concepts, and 
ability to apply them appro.priately) required to perform jo6 tasks must be 
inferred, and consequently domain reJerenced testing is used for assessment 
of achievement in cognitive portions* of the training program objectives. 
, In domain referenced. testing the* goal is to create an extensive pool 

*^«f Items which represents, in miniature, the basic characteristics of some 
fmpcrtanl n^rf the criginnl universe of knowled*ge domain (Hively, 1974). 
. /..doi-.ain must bo rp.p«!bln of being described ve^y specifically both in terras 
of " content: and format. Th.-^ raj or advantage of donain referenced testing is 
th^t it ^ITowe nstim tes of a participant's "lev*! o5 functioning" x-rom a • 
/ small par>ple of it'^'^'J or ther percentage of the total tasks of a Specified 
typr. vhJch woiu/1 ha anpw^r^^d corr^.ctly. , Tb*?. raii.^fcility of' the test i« the 
accuracy with irhich the probabilities of correct performance can be estimated. 



Validity can be assessed by, logical analysis^of the domain definition, the 

. * * * 

* * « 

item generation scheme, a.nd thfe individual test items (Millman, 197A). ^ 1 

The domain referred to for the purposes of the model i§ an educational 

objective. Consequently, in Jcvelopins assu^ssment procedures for flxiy objective 

with a cognitive component, an attempt should be made to generate a large set 

of test items which represent ,the pool for thcu domain. The. number of itemb 

generated is limit,ed by practical constraints^rcost and computer space. For 

the program developed in the model*s first qppiicat ion 100 test quest Ions 

for each objective were generated. A domain oj: objective sfiould be period- 

ically reviewed to determine their continued relevance^ For example, a 

* ' • * 

training objective may state ^hat a special education adnjinistrator rtust be 

* * 
cognizant of the requirements of due process, but a change in law or regulation 

may alter specific due process procedures which the director must. follow. 

Act-ual testing of the model u^der a domain referenced measurement method 

is done 'by means of a Bayesicn Instructional Testing System (Special Education 

Administratian Training Program, 1974) which is a ranciom selection of those 

items which measure the objective. The items selected for inclusion in pre- 

tests cover all objectives being assessed and are randomly mixed. , An estimate 

is made of the criterion l.ey^l (e.g. 80 percent correct) which constitutes 

.mastery of each objective (domain), and instruction is provided in thd.se 

domains where any participant falls b^low the criterion level. Post tests 



are developed individually for each participant and consist of items randomly 

selected from each domain in whicli^ instruction has been provided. A. separate 

post test should be developed for/ each objective to permit each participant 

« * 

to be tested as soon as he completes the .course and to allow repeated and 

different post tests on each ^objective until the mastery criterion is reached. 

/ 
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The criterion level for mastery is initially set* .at an arbitrary 

X* • * 'J 

• level, based on the judgment of program staff. After data on domaJCn 

referenced test performailce and on performance assessment ate available, 
, scores on the two assessment's are compared to determiae the l^vel of 
' achievement, on the domain refererited test necessary to predict pass ratings 
on the simulations. That level then becomes the validated mastery criterion 
for^ the domain- referenced test. « • * * , 

Since participants are tested on only a small .fraction of" the itfems 

» " • \\ 

which measura achievement of each objective, the reliability of a 'domain 

referenced testing procedure is dependent upon the probability that each 

participant s score on the items to which he/she responds represents the 

score he/she would attain on the entire (infinite) set of items in that 

domain. Bayesian statistical proced^ures (Novick and Lewis, 1974;. Novick- . 

and Jackson, 1974) are used to prescribe the length of the test a participant 

§h6ul<f receive and to determine the criterion level which approximates the 

mastery criterion for the entire /domain. 

Prior to. testing, an arbitrary estimate is made of the probability that 

participants wall achieve at the mastery, criterion level, which is xised to 
#. ^ 

determine pretest length and the passing score for each objective. Once a 
test ^ has been administered, , information is combined in a straightforward 
Baj[esian procedure using the'het.ci distribution to obtain prior estimates for 
the first post tiest. This procedure .continues unt'il the estimate of the* 
pfobability that the participant performs at the mastery level is sufficiently 
high to consider him/her passed. "Sufficiently high" is deterinined by the 
loss rat;io for an objective; e.g. a loss ratio of^ 1.^ indicates that '.the loss , 
associated with incorrectly passing a participant who has not reached fHe 
criterion level is one and one-half times greatef than the loss associated 



^with inco-^rectly "failing" a student who Has reached or exceeded the criterion 

level* For example, an eight item post test on an objective with a passljig 

score of 6 (75 perceA) might be recommended vh^n the mastery critei:ion is 

70 pei;cenf ^nd the los's ratio is 1»5 (the score required on the test is 

higher th&n the mastery criterion because of the short length of the test 

and because the loss ratio Is, more than 1)» ■ ' ^ 

All cognitive assessment, information is recorded and scored on computer, . 

and the system developed (Bayesian Instructional Test System) for use with :5 

the model contains programs and^ disc storage files whicU contain the item 

pool; maintain the statu3 of individual participants in the training, project; 

select, print, and. score pre aad post tests for each participant; .and maintain 

an ongoing statistical, summary of par.ticipant progress through the training . 

program. There are nine computer programs for these purposes: 

. 1» Creation of the mas t«^r item' fi.le, ^including add'^tions, 

modific'ati'ons, and deletions/ ^ ' ' 

2* Recording of criterion'.ieveis and' loss ratios for each 

. objective (for a given group of participants, thip ^ , . 

information, is fixed) -r ^ ' 



3... Providing initial information on each participant which ^ 
will be used in 'later programs ^including, estimates^ o'f ' 
the probability that a 'participant has achieved the 
criterion level, participant training and experience ' * 
^ data, etc. ) , ^ ' 

4, Determination of the number of items to be included la 
the pretesc and random selection of items from the master 
file. * ^ ' • , 

5^ Determining format and printing a copy of the batch pretest 
for each participant. 

6. 'Scoring the pretest and updating' the files for each partici- 
^ pant tincluding a determination of the need for instruction 
•and for •a post test based oh a revised achieve^ment estimate). 

7» Examination of the participant's status and selection of 
items for. a first post t&st* ; 
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8» Printing a post test and answer shefe^for any post test. 

9. -Scoring post tests,, updating the information file on 
each •participant, and generating a new post t^st for 
each objective riqt passed • (A participant 'may take up. 
to eight post teste per objecti^ve.) 

Service Delivery Systems" ' \ 

The manner in which uh^ characteristics appear in actual operation 
can perhaps be most easily understood by -listing the cycle of Activities 
Included in the .model as inftlaily implemented* However, the purpose of" 
the .present section Is merely to illustrate a special education adminis- 
trator's activities as a participant in this program, not to indicate 
that this is the only delivery system for the model***' Specific needs of 
each application will determine the delivery system used. This can be 
eithet in-service (continuing education) or pre-seivice (entry level educa- 

tion) . A sequential representation 5f his participation and the service 

* 

delivery system are shown as Figure 7. . 

^ . s 

" _ Insert Figure 7 here ^ 

Selection . In the, illustration Minnesota special ^^education directors' 

wer^ eligible to participate in the program if they indicated Interest and 

met the initial selection criteria^ Selection criteria included Ijknitcd 

experience^ as a director of special education, (less t:hau three years), 

little or no formal training in educational administration, and a capacity 

lljnit of 25 participants. These criteria were established to maximize 

immediate impabt of the project in its formative phases while in the future, 

participation may be open to all inte.rested/ 

Program planning . Following acceptance Into the illustrative program, 
• * » * 

U participant's first activities consisted of an assessment of his individual 
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needs, ^irs^, participants were given a cognitive domain referenced test 
covering knowledge and application of facts, procedures, and' concep^s^jfor 
all objectives in each of the three curriculum areas which had been identified: 
fiscal management, personnel management, and special education program develop- 
ment. The format'^ of the test was a series of approximately 30b multiple 
choice, true-false, and .other questions in each of the three curriculum areas. 
If available information, such as results of prior training, indicated that 
a partlcdtpant had already mastered an area, the test could be shortened ^ 

^accordingly. Jest results were analyzed to determine areas in which particl- 
pants had or^^id not have requisite knowledge, using predetermined criteria* • 
Areas of deficit for "each participant became his training objectives- Criteria 
for adequate cognitive levels were established by correlating domain referenced 
score's with performance assessment .results. 

In addition to the domain referenced test,^ assessment included rating 
of each participant's performance using simulations of tasks necessary to 
the position and self-reporting of on-the-job performance. Simulations were 
rated iudependengly by a panel of judges, and the majority opinion became 
each participant'-: score. As with the domain referenced test, performance 
areas were compared tjlth the predetermined criteria, deficits were determined, 
and the results were used tp formulate individual training objectives. 

Cognitive pretests were administered via mail and were returned and 
analyzed prior to the performance assessment, PerforSiance simulations were 
conducted in a workshop setting. The workshop was not only a convenient 

^yehicl^' for performance assessment but also provided an opportunity for 
initial instruction in the objectives for the participants and for prograrir 

y 

planning. with staff. In addition, the workshop provided orientation to the 
field experiences in which participants were to engaged 
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Instri^tlon , Each participant's program in the field required completion 
of course materials appropriate to his/her 'needs, provided periodic con- 

y 

sultant assistance in improving performance*, and allowed opportunities for 

» 'J 

sm^ll gr.oup interaction on problem-solving exercises t 

First, a participant was sent a set of apr^opriate field materials and 

<« 

activities for each objective in which the pretest showed his/her performance 
to be below the' criterion level. Course materials included presentation of 
concepts, source materials, and alternative suggestions for methods of 
implementing the concept* Participants then completed an exercise demonstrating 
their ability to Ijnplement the concept as it applies to their job; in many 
cases, course exercises were tasks which must be done on the job in any 
event (such as developing a child study, subsystem) ♦ Exercises were assessed 
by authors of th^ course materials who based their judgments on evidence that 
a participant had correctly understood the concept and application of the 
concept was appropriate to a participant's situation. Exercises were rated 
"a*ccep table," "incomplete," or "unacceptable," and comments were included. 
This part of the field experience -was conducted by mail. 

Course authors' critiques were reported not only to participants but 
also the participant's field consultant. Field consultants were parsons who 
are expert, in a particular curriculum area (e.g« perspnnel, fiscal, and 
program development), and who worked in the same, geographic area of the 
State as a group of participants. In many cases field consultants were 
persons with whom participants were likely to have ongoing communications 
after the training program was completed* Each field consultant met monthly 
with a group of participants who were studying in similar curriculum areas. 
During these day-long meetings, assignments were reviewed and problems , 
were discussed^ the primary role of the field consultant was to assist 

30 
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each participant in maximizing his/her achievement in both cognitive and 
performance areas, rather than to evaluate a participant's performance. 
Field consultants did have an evaluative role,, but their reporting of 
problems encountered vith instijuction .provided a valuable source of feed- 
back for course authors and project staff in order to improve instructional 
materials. Participant evaluations of each instructional package and post 
test scores were also utilized to determine areas of improvement in the 
course materials. 

Following rev^.ew with assistance by a field consultant > participants 
could modify or repeat course activities as needed, and' the same prQcediire— 
assessment by course authors and review by participant and field consultant-- 
waQ repeated until adequate performance was attained. This cycle of input 
from course materials, practice or exercises included as part of the course 
materials, feedback pn adequacy of performance, and assistance in improving 
performance continued thrQughout the training .program. ' - * 

' Evaluation * After a participant satisfactorily completed^ instruction 
in a curriculum area, the assessment process was repeated using post test 
versions of both the domain referenced test and performance simulation. The 
post instructional performance assessment given after instruction included 
items directed only at the areas in which a parti .ipant was rated deficient 
on the pretest. 

Certification/ credit . In the exainple, administrative certification was 

usually circumvented for entry into the position of special education director. 

*« 

It is possible that* training offered (i.e. competencies attained) under such 

'* ' ' 

a program could be directly applied toward future certification as certif ica- 

tlon requirements are reviewed. It might also be ^that competencies certified 
through a training program could be used to meet continuing education require- 
ments established by the state, local, district, or other agency." 
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Time Parameters for Developmental Phases 



Use of syst^s^models helps to clarify the logical, strucWre of a 
procedure since they are relatively independent of content. Systems models 
are also independent of time. However, based upon our, experience, it may 
be useful to add some estimates of the amount of time. which should be allowed 
for development of each component of a training program using the field 
.centered, competency based^model. 

The amount of time required for initial 'determination of the population 

to be trained will vary«with the method used and with the extent of documenta- 

♦ 

txon of n?ed required by particular funding authorities* Such activities 
are usiially done before a training model is selected. Thus, time estimates 
for this component are not included here. 

^However, for deyelopment of the remaining components of the moder a 
minimum of one year must be allowed. The amount of .staff time and other 
resources^ which must be deployed during that year will vary with the extent 
to which development procedures and content (objectives, item pool for domain 
•referenced testing, and iikStructional materials) can be used or adapted. 
Thus, less effort will usually be required to develop a preparation program 
for a position for which such a program has been developed in another stcXte 
using the model than would be required to develop a comparable program for 
a position for which a program has not been previously developed using the 
model* One might also project that less effort will be required to develop 
an administ;ca'tive education program than, one for teachers or other direct 
service providers. The procedures, however, i^ill be applicable in any case. 

A one year development period seems necessary for competency identlftca- 
tion, due to the inclusion of both identification and validation procedures 
in the development phase. Given favorable conditions, gosl analysis and job 
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analysis can both be ^ accomplished in 90 days. However^ an anthropological 
field study to validate initially identified competancieB requires an entire 
year; and, if possible, additional time should be allowed for thorough data 
analysis. 

The Job analysis and goal analysis provide the training objectives 

which ^are necessary input into both the preparation of instruction and develop 

ment of assessment components* Once objectives are known, course authors can 

be selected. and materials preparation begun. If some use can be made of 

SEATP materials or if instructional materials for objectives identified as ^ 

high priority are readily available, 'instructional preparation for a year*g 

instruction may be done in six months. If instruction is likely to be 
, " it 

sequential, some instructional preparation can continue while initial course 

work is conducted. 

The major tasks in developing as8essmentj.procedures> if Bayesian Testing 
System computer programs are u§ed, are preparing an item pool and dweloping 
simulated or on-the-job performance assessment procedures. If many items in 
previously developed item pools ar,e applicable to a proposed educat.''>n 
program, the task may be accomplished in eight months. If the Entire pool 
must be developed* then a minimum of a year (after training objectives have 
been determined) must be allowed.' Development of performance assessment 
procedures also varies with the extent to which existing simulation materials 
and other tools can be employed. An emphasis on performance ratings rather 
than the domain referenced test procedure would result in proportionately less 
tijne being spent in instrument development (^nd more in administration of the 
performance assessment) than is the case for cognitive assessment. 
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Sviimnary 

This field centered, competency based educational model considered 
in this B^per is an ongoing part of a pro*gram for the special education 

administrators in Minnesota. It is believed to have potentiaj. for wider 

{ 

\ 

application since it repres.ents what the writers believe to be a general 
^lodel. . - 
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